Summary
Background: Online medical knowledge repositories such as MEDLINE and The Cochrane Library are
increasingly used by physicians to retrieve articles to aid with clinical decision
making. The prevailing approach for organizing retrieved articles is in the form of
a rank-ordered list, with the assumption that the higher an article is presented on
a list, the more relevant it is.
Objectives: Despite this common list-based organization, it is seldom studied how physicians
perceive the association between the relevance of articles and the order in which
articles are presented. In this paper we describe a case study that captured physician
preferences for 3-element lists of medical articles in order to learn how to organize
medical knowledge for decision-making.
Methods: Comprehensive relevance evaluations were developed to represent 3-element lists of
hypothetical articles that may be retrieved from an online medical knowledge source
such as MEDLINE or The Cochrane Library. Comprehensive relevance evalua tions asses
not only an article’s relevance for a query, but also whether it has been placed on
the correct list position. In other words an article may be relevant and correctly
placed on a result list (e.g. the most relevant article appears first in the result
list), an article may be relevant for a query but placed on an incorrect list position
(e.g. the most relevant article appears second in a result list), or an article may
be irrelevant for a query yet still appear in the result list. The relevance evaluations
were presented to six senior physi cians who were asked to express their preferences
for an article’s relevance and its position on a list by pairwise comparisons representing
different combinations of 3-element lists. The elicited preferences were assessed
using a novel GRIP (Generalized Regression with Intensities of Preference) method
and represented as an additive value function. Value functions were derived for individual
physicians as well as the group of physicians.
Results: The results show that physicians assign significant value to the 1st position on
a list and they expect that the most relevant article is presented first. Whilst physicians
still prefer obtaining a correctly placed article on position 2, they are also quite
satisfied with misplaced relevant article. Low consideration of the 3rd position was
uniformly confirmed.
Conclusions: Our findings confirm the importance of placing the most relevant article on the 1st
position on a list and the importance paid to position on a list significantly diminishes
after the 2nd position. The derived value functions may be used by developers of clinical
decision support applications to decide how best to organize medical knowledge for
decision making and to create personalized evaluation measures that can augment typical
measures used to evaluate information retrieval systems.
Keywords
Organization of medical evidence - physician preferences - rank-ordered lists - evidence-based
medicine - information retrieval - decision support systems - clinical
